A Peek Into the Memory of T5: Investigating the Factual Knowledge Memory in a Closed-Book QA Setting and Finding Responsible Parts

Recent research shows that Transformer-based language models (LMs) store considerable factual knowledge from the unstructured text datasets on which they are pre-trained. The existence and amount of such have been investigated by probing pre-trained Transformers to answer questions without accessing any external context or (also called closed-book question answering (QA)). However, this is spread over parameters inexplicably. parts model most responsible for finding an only a unclear. This study aims understand T5 reaching in QA setting. Furthermore, we introduce head importance scoring method compare it with other methods three datasets. We investigate important looking inside attention heads novel manner. also why some more critical than others suggest good identification approach. demonstrate retaining through series pruning experiments. roles encoder decoder